Fix $bulk-delete bugs with custom search parameter #4964

tarunmathew12 · 2025-05-01T21:56:01Z

Description

The PR will address an issue of the search parameter status going out of sync after $bulk-delete removes search parameter resources.

Related issues

Addresses [issue #121676].
Bug 121676: Hard delete custom search parameter not cleaning up all the data

Testing

Tested manually and also through adding E2E test.

FHIR Team Checklist

Update the title of the PR to be succinct and less than 65 characters
Add a milestone to the PR for the sprint that it is merged (i.e. add S47)
Tag the PR with the type of update: Bug, Build, Dependencies, Enhancement, New-Feature or Documentation
Tag the PR with Open source, Azure API for FHIR (CosmosDB or common code) or Azure Healthcare APIs (SQL or common code) to specify where this change is intended to be released.
Tag the PR with Schema Version backward compatible or Schema Version backward incompatible or Schema Version unchanged if this adds or updates Sql script which is/is not backward compatible with the code.
When changing or adding behavior, if your code modifies the system design or changes design assumptions, please create and include an ADR.
CI is green before merge
Review squash-merge requirements

Semver Change (docs)

Patch|Skip|Feature|Breaking (reason)

test/Microsoft.Health.Fhir.Shared.Tests.E2E/Rest/BulkDeleteTests.cs

src/Microsoft.Health.Fhir.Core/Features/Search/Registry/SearchParameterStatusManager.cs

src/Microsoft.Health.Fhir.Core/Features/Search/Parameters/SearchParameterOperations.cs

tarunmathew12 · 2025-05-27T15:57:26Z

/azp run

azure-pipelines · 2025-05-27T15:58:12Z

Azure Pipelines successfully started running 1 pipeline(s).

tarunmathew12 · 2025-05-27T18:45:03Z

/azp run

azure-pipelines · 2025-05-27T18:45:47Z

Azure Pipelines successfully started running 1 pipeline(s).

src/Microsoft.Health.Fhir.Core/Features/Definition/SearchParameterDefinitionManager.cs

src/Microsoft.Health.Fhir.Core/Features/Persistence/IDeletionService.cs

LTA-Thinking

I don't believe the response from the deletion service can be a list of resource wrappers due to memory limitations.

tarunmathew12 · 2025-05-30T21:43:42Z

I don't believe the response from the deletion service can be a list of resource wrappers due to memory limitations.

We have reverted it back to returning the stats dictionary. We have also added an optional parameter to DeleteMultipleAsync to store deleted search parameters.

tarunmathew12 · 2025-06-04T17:05:56Z

/azp run

azure-pipelines · 2025-06-04T17:06:39Z

Azure Pipelines successfully started running 1 pipeline(s).

tarunmathew12 · 2025-06-05T16:17:54Z

/azp run

azure-pipelines · 2025-06-05T16:18:09Z

Azure Pipelines successfully started running 1 pipeline(s).

Fixing $bulk-delete bugs.

aa190d4

tarunmathew12 added this to the CY25Q2/2Wk02 milestone May 1, 2025

tarunmathew12 requested a review from a team as a code owner May 1, 2025 21:56

github-advanced-security bot found potential problems May 1, 2025

View reviewed changes

test/Microsoft.Health.Fhir.Shared.Tests.E2E/Rest/BulkDeleteTests.cs Fixed Show fixed Hide fixed

test/Microsoft.Health.Fhir.Shared.Tests.E2E/Rest/BulkDeleteTests.cs Fixed Show resolved Hide resolved

tarunmathew12 added 6 commits May 1, 2025 15:35

Fixing exception types.

62daa03

Addressing bot's comment

53acd9b

Removing the redundant grouping/summing.

d4ae388

Merge branch 'main' into personal/v-tmathew/bulk-delete-bug

21b8701

Fixing some test failures.

179274e

Adding some debug statements

6917021

github-advanced-security bot found potential problems May 5, 2025

View reviewed changes

test/Microsoft.Health.Fhir.Shared.Tests.E2E/Rest/BulkDeleteTests.cs Fixed Show fixed Hide fixed

test/Microsoft.Health.Fhir.Shared.Tests.E2E/Rest/BulkDeleteTests.cs Fixed Show fixed Hide fixed

tarunmathew12 added 5 commits May 5, 2025 15:55

Debugging flaky tests.

4536421

Merge branch 'main' into personal/v-tmathew/bulk-delete-bug

d4a5874

More changes for investigation the flaky tests.

af3484b

More change for flaky test investigation.

ff8250c

Fixing the flaky tests.

19439a2

github-advanced-security bot found potential problems May 6, 2025

View reviewed changes

test/Microsoft.Health.Fhir.Shared.Tests.E2E/Rest/BulkDeleteTests.cs Outdated Show resolved Hide resolved

Add more code to ensure each operation passes.

8e982d9

github-advanced-security bot found potential problems May 7, 2025

View reviewed changes

test/Microsoft.Health.Fhir.Shared.Tests.E2E/Rest/BulkDeleteTests.cs Show resolved Hide resolved

test/Microsoft.Health.Fhir.Shared.Tests.E2E/Rest/BulkDeleteTests.cs Fixed Show fixed Hide fixed

tarunmathew12 added 4 commits May 8, 2025 10:17

Merge branch 'main' into personal/v-tmathew/bulk-delete-bug

06beb09

Renaming helper functions.

8e54cb9

Investigating flaky tests.

942c800

Adding debug statements.

95e9407

github-advanced-security bot found potential problems May 8, 2025

View reviewed changes

test/Microsoft.Health.Fhir.Shared.Tests.E2E/Rest/BulkDeleteTests.cs Show resolved Hide resolved

tarunmathew12 added 2 commits May 12, 2025 14:00

Merge branch 'main' into personal/v-tmathew/bulk-delete-bug

3bd595d

Investigating flaky tests.

026b92a

tarunmathew12 added 4 commits May 20, 2025 13:27

Adding retry on SP update by bulk delete.

17dc3da

Merge branch 'main' into personal/v-tmathew/bulk-delete-bug

66d833e

Adding more logging for debugging the flaky test.

ea18303

Fix delete search parameter function.

220dacf

github-advanced-security bot found potential problems May 21, 2025

View reviewed changes

src/Microsoft.Health.Fhir.Core/Features/Search/Registry/SearchParameterStatusManager.cs Fixed Show fixed Hide fixed

src/Microsoft.Health.Fhir.Core/Features/Search/Registry/SearchParameterStatusManager.cs Fixed Show fixed Hide fixed

tarunmathew12 added 5 commits May 22, 2025 09:24

Merge branch 'main' into personal/v-tmathew/bulk-delete-bug

b205575

fixing merge conflict.

6c4a4c6

Removing the log statements for debugging.

228156d

Fixing merge conflicts.

98a8a1a

Cleaning up redundant code.

5c9c27f

github-advanced-security bot found potential problems May 22, 2025

View reviewed changes

src/Microsoft.Health.Fhir.Core/Features/Search/Parameters/SearchParameterOperations.cs Show resolved Hide resolved

Addressing bot comment.

59cb78a

Fixing merge conflicts.

2ec5767

LTA-Thinking reviewed May 30, 2025

View reviewed changes

src/Microsoft.Health.Fhir.Core/Features/Definition/SearchParameterDefinitionManager.cs Outdated Show resolved Hide resolved

LTA-Thinking reviewed May 30, 2025

View reviewed changes

src/Microsoft.Health.Fhir.Core/Features/Persistence/IDeletionService.cs Outdated Show resolved Hide resolved

LTA-Thinking requested changes May 30, 2025

View reviewed changes

tarunmathew12 added 2 commits May 30, 2025 14:25

Merge branch 'main' into personal/v-tmathew/bulk-delete-bug

9706ecc

Addressing RB's comments.

373b00e

tarunmathew12 added 3 commits June 2, 2025 09:11

Merge branch 'main' into personal/v-tmathew/bulk-delete-bug

5ece706

Fixing a UT failure.

6144df2

Fixing test failures on bulk-delete soft-delete.

a771700

Fix $bulk-delete bugs with custom search parameter #4964

Are you sure you want to change the base?

Fix $bulk-delete bugs with custom search parameter #4964

Uh oh!

Conversation

tarunmathew12 commented May 1, 2025

Description

Related issues

Testing

FHIR Team Checklist

Semver Change (docs)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tarunmathew12 commented May 27, 2025

Uh oh!

azure-pipelines bot commented May 27, 2025

Uh oh!

tarunmathew12 commented May 27, 2025

Uh oh!

azure-pipelines bot commented May 27, 2025

Uh oh!

Uh oh!

Uh oh!

LTA-Thinking left a comment

Choose a reason for hiding this comment

Uh oh!

tarunmathew12 commented May 30, 2025

Uh oh!

tarunmathew12 commented Jun 4, 2025

Uh oh!

azure-pipelines bot commented Jun 4, 2025

Uh oh!

tarunmathew12 commented Jun 5, 2025

Uh oh!

azure-pipelines bot commented Jun 5, 2025

Uh oh!

Uh oh!